An Achievement Test for Knowledge-Based Systems: QUEM

This paper describes QUEM, a method for assessing the skill level of a knowledge-based system based on the quality of the solutions it produces. QUEM is demonstrated by using it to assess the performance of a particular knowledge-based system, P. QUEM can be viewed as an achievement or job placement test given to know ledge-based systems to help system designers determine how the system should be used, and in what capacity by what level of users. In general, it is difficult to find useful metrics for assessing a system's overall performance. Most literature on evaluation deals with validation, verification and testing in which the primary concern is the correctness and consistency in the databases and rulebases. However, these properties alone may not be sufficient to determine how well a system performs its task. QUEM allows software developers to assess their system's performance by constructing a skill function based on human performance data that relates experience and solution quality. QUEM can be used to gauge the experience level of an individual system, compare two systems, or compare a system to its intended users. This represents an important advance in quantitative measures of over-all system performance that can be applied to a broad range of systems.